Three-dimensional audio systems

ABSTRACT

A three-dimensional sound generation system includes one or more processors of a computing device, including capability to receive sound tracks, each of the sound tracks comprising one or more sound sources, each of the one or more sound sources corresponding to one or more respective sound categories, receive or determine a first configuration in a three-dimensional space, the first configuration comprising a listener position and a computing device location relative to the listener position, determine a second configuration comprising a change to at least one of the listener location or the computing device location relative to the listener position, generate, using the one or more sound tracks and the second configuration, one or more channels of sound signals, and provide the one or more channels of sound signals to drive one or more sound generation devices to generate a three-dimensional sound field.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Application 17/568,343 filedon Jan. 4, 2022, which is a continuation of U.S. Application No.17/227,067 filed on Apr. 9, 2021, which claims the benefit of thefollowing patent applications: U.S. Provisional Application Ser. No.63/008,723, filed on Apr. 11, 2020; and U.S. Provisional ApplicationSer. No. 63/036,797, filed on Jun. 9, 2020. The contents of theabove-mentioned applications are hereby incorporated by reference intheir entirety.

TECHNICAL FIELD

The present disclosure relates to the generation of three-dimensionalsound, and in particular to systems and methods for capturing andprocessing mixed sound tracks into separate sound types and thenapplying transfer functions to the separated sound to generatethree-dimensional sound that contains spatial information about thesounds sources to recreate a three-dimensional (3D) sound field that hasbeen configured by users.

BACKGROUND

Billions of people listen to music worldwide, but most listeners mayonly have access to music in a mono or stereo sound format. Stereo is amethod of sound reproduction that may use multiple independent audiochannels played using two or more speakers (or headphones) so that thesound from the speakers appears to be coming from various directions, asin natural hearing. However, stereo sound usually refers to just twoaudio channels to be played using two speakers or headphones. Moreimmersive sound technologies like surround sound need to record and savemultiple sound tracks (e.g., 5.1 or 7.1 surround sound configurations),and the sound must be played through an equivalent number of speakers.In any case, each of the audio channels or sound tracks consists ofmixed sound from multiple sound sources. Therefore, stereo sound isdifferent from “real” sound (e.g., a listener in front of a stage at aconcert) because spatial information regarding the individual soundsources (e.g., instruments and vocals) is not reflected in the sound.

With two ears, a person may perceive spatial information and hear “real”three-dimensional (3D) sound as binaural sound (e.g., sound representedby a left ear and a right ear), such as how music is perceived by twoears in a music hall, theater or at a sporting event at a stadium orarena. However, as noted above, today’s music technology usuallyprovides only mono or stereo sound without spatial cues or spatialinformation. For this reason, music and other sounds may be experienceddifferently and often more enjoyably in theaters, arenas, and musichalls than it is through headphones or earbuds or on loudspeakers oreven on multiple-channel, multiple-loudspeaker surround systems.Currently, the generation of 3D sound may be accomplished, for example,by many loudspeakers mounted on the walls of a movie theater with eachloudspeaker being driven by a separate sound track recorded duringmanufacturing of a movie. However, this kind of 3D audio system may bevery expensive and cannot be realized in mobile devices as an app(application software) or even in most home theater or in-carconfigurations. Therefore, in today’s music and entertainment industry,most of music or other audio data is stored and played as mono or stereosound, where all sound sources, such as vocals and different kinds ofinstruments, are pre-mixed into just one (mono) or two (stereo) soundtracks.

Most audio/sound from a video conferencing device, such as a computer,laptop, smartphone, or tablet, is in mono sound. Although on a displayscreen, a user (e.g., an attendee or participant), may see all attendeesof the conference in separate windows, the audio is usually only onechannel mono with a narrow bandwidth. Using video of each of thedifferent attendees, a virtual conference room may be accomplished, butthe audio component cannot match the video component because it does nothave the 3D sound which is necessary for providing a more accurate(e.g., spatially) virtual reality sound experience. Furthermore, whentwo attendees have similar sounding voices, the user may not be able todistinguish between voices when they are talking at the same time oreven separately. This may happen, for example, when the user is watchingshared documents on another screen or video window while the user is notlooking at the attendees’ faces. The problem may be even worse when moreattendees are in a video conference, such as a remote learningclassroom. The user may need spatial information, like 3D sound, to helpidentify which attendee is speaking based on the conference sound alone.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure will be understood more fully from the detaileddescription given below and from the accompanying drawings of variousembodiments of the disclosure. The drawings, however, should not betaken to limit the disclosure to the specific embodiments, but are forexplanation and understanding only.

FIGS. 1A-1B illustrate systems for generating three-dimensional sound,according to implementations of the present disclosure.

FIGS. 2A-2B illustrate a spatial relationship between a sound source anda listener in a three-dimensional space and a selection of filters forgenerating 3D sound that reflects the spatial relationship, according toimplementations of the present disclosure.

FIG. 3 illustrates a system for training a machine learning model toseparate mixed sound tracks, according to an implementation of thepresent disclosure.

FIG. 4 illustrates a system for separating and filtering mixed soundtracks using transformed domain sound signals, according to animplementation of this disclosure.

FIGS. 5A-5E illustrate original mixed sound in waveform and spectrogramand the mixed sound separated into vocal, drum, base, and other sound,respectively, according to implementations of the present disclosure.

FIG. 6 illustrates far-field voice control of a 3D binaural music systemwith music retrieval by voice and sound separation, according to animplementation of the present disclosure.

FIGS. 7A-7D illustrate a GUI for user configuration of 3D sound withselected listener positions inside a band formation (7A-7C) and in thefront of the band formation (7D), respectively, according toimplementations of the present disclosure.

FIG. 8 illustrates a system for generating 3D sound with a microphonearray, according to an implementation of the present disclosure.

FIGS. 9A-9B illustrate beam patterns for a 3D microphone and a 3Dmicrophone array with spatial noise cancellation, respectively,according to implementations of the present disclosure.

FIG. 10 illustrates a conference or virtual concert system forgenerating three-dimensional sound, according to implementations of thepresent disclosure.

FIG. 11 illustrates a virtual conference room displayed for a GUI of aconference system for generating three-dimensional sound, according toimplementations of the present disclosure.

FIG. 12 illustrates a method for generating three-dimensional sound,according to an implementation of the present disclosure.

FIG. 13 illustrates a method for generating three-dimensional sound,according to an implementation of the present disclosure.

FIG. 14 illustrates a block diagram of hardware for a computer systemoperating in accordance with one or more implementations of the presentdisclosure.

FIG. 15 illustrates a loudspeaker distribution in a sound bar accordingto an implementation of the present disclosure.

FIG. 16 illustrates a loudspeaker distribution in a sound bar withseparated stereo plus sound according to an implementation of thepresent disclosure.

FIG. 17 illustrates a loudspeaker distribution for a TV or a movietheater according to an implementation of the present disclosure.

FIG. 18 illustrates a loudspeaker distribution for a TV or a movietheater according to another implementation of the present disclosure.

FIG. 19 illustrates a speaker matrix deployed with a TV or a movietheater according to an implementation of the present disclosure.

DETAILED DESCRIPTION

Described herein are three-dimensional (3D) configurable soundstageaudio systems and applications and implementations. A three-dimensional(3D) sound field refers to sound that includes discrete sound sourceslocated at different spatial locations. The 3D soundstage is the soundrepresenting the 3D sound field. For example, soundstage music may allowa listener to have an auditory perception of the isolated locations ofinstruments and vocal sources when listening to a given piece of musiceither through earphones, headphones, or loudspeakers. In general, the3D soundstage may have embedded cues for the listener’s perception ofthe spatial information. The soundstage may also be configurable so thatit may be configured by the listener, a DJ, software, or audio systems.For example, the location of each instrument in the 3D sound field maybe moved while the listener’s location in the 3D sound field may bedynamic or static at the location of a preferred instrument.

In order to listen to or to play the 3D soundstage a listener may usebinaural sound represented by two tracks, one for the left ear and onefor the right ear, with embedded cues for listener perception of spatialinformation associated with sound sources. Binaural sound may beexperienced as 3D sound (e.g., as if coming from different locations)through earphones, headsets or other such devices. Alternatively, direct3D sound may be used to play the 3D soundstage. In direct 3D sound, thesound is played from a group of loudspeakers located in different 3Dlocations (e.g., corresponding to desired locations for individual soundsources in the 3D sound field). Each loudspeaker may play one isolatedsound track, e.g., one speaker for drum and another for bass. Thelistener may hear the 3D sound field from the loudspeakers directlysince they are at different locations in a real world 3D sound field. Inboth the binaural and direct 3D sound use cases, the listener’s brainmay perceive the 3D sound field and may recognize and track the discretesound sources like in the real world, which may be referred to asacoustic virtual reality throughout the present disclosure.

Furthermore, another way to achieve the 3D sound field may be to recordbinaural sound directly with a specialized binaural/3D microphone. Mostexisting binaural microphones are just a dummy human head withmicrophones installed in the ears which may be too big in the sizeand/or too expensive for many applications. Accordingly, describedherein is a 3D microphone that may have a small form factor by using anarray of very small microphones and signal processing technology. Thissmall form 3D microphone may be used with any handheld recording devicesuch as a smartphone or tablet. The output of the sound captured by the3D microphone may be presented as binaural, stereo, or multi-trackrecordings, with one track for each spatial direction associated with asound source for the 3D sound field.

Also, in the present disclosure, three techniques are described toenhance the signal-to-noise ratio (SNR) of audio signals as follows.Noise reduction is the process of reducing the background noise in anaudio channel based on temporal information, such as the statisticalproperties between signal and noise or the frequency distributions ofdifferent kinds of signals. A microphone array uses one or multipleacoustic beam patterns to enhance the sound coming from one beamdirection while canceling the sound coming from outside the beamdirection. An acoustic echo canceller (AEC) uses one or more referencesignals to cancel the corresponding signals mixed in the microphonecaptured signals. The reference signal(s) is/are correlated to thesignal(s) which the AEC will cancel.

Systems

FIGS. 1A-1B illustrate systems 100A and 100B for generatingthree-dimensional sound, according to implementations of the presentdisclosure. Systems 100A and 100B may be standalone computer systems ora networked computing resources implemented in a computing cloud.

Referring to FIG. 1A, system 100A may include a sound separation unit102A, a storage unit 104A for storing a plurality of filters such ashead related transfer function (HRTF) filters, all-pass filters, orequalization filters, a signal processing unit 106A, and a 3D soundfield configuration unit 108A with a graphical user interface (GUI) 110Afor receiving user input. For the conciseness of discussion, the filtersin the following are referred to as HRTF filters although it isunderstood that the filters can be any types of suitable filtersincluding all-pass filters or equalizer filters. The sound separationunit 102A, the storage unit 104A and the 3D sound field configurationunit 108A may be communicatively coupled to the signal processing unit106A. Signal processing unit 106A may be a programmable device that maybe programmed to implement three-dimensional sound generation accordingto configurations received via the GUI 110A presented on a userinterface device (not shown).

In the example of FIG. 1A, the input to sound separation unit 102A isoriginal mixed sound tracks of mono or stereo signal or audio, while theoutput from signal processing unit 106A is 3D binaural audio for leftand right ears, respectively. Each of the input of mixed tracks orchannels may first be separated into a set of separated sound tracks(e.g., for one corresponding sound source that may be associated withone or more sound types) by the sound separation unit 102A, where eachtrack represents one type (or category) of sound, for example, vocal,drums, bass, or others (e.g., based on the nature of the correspondingsound source).

Each of the separated sound tracks may then be processed by signalprocessing unit 106A using a pair HRTF filters from storage unit 104A tooutput two audio channels representing left and right ear channels,respectively, for each separated sound track. In one implementation, theabove-noted process may be performed in parallel for each of the inputmixed sound tracks.

Each HRTF filter (e.g., a pair of left and right HRTF filters 200B ofFIG. 2B described below) may be associated with a point on the grid inthe three-dimensional space (e.g., the HRTF filters may be stored as amesh of grid points in a database) and each of the grid points may berepresented by two parameters: azimuth angle θ and attitude angle γ(e.g., 202B and 204B of FIG. 2B respectively). The mesh of HRTF filters(e.g., 200B) may be an array of pre-computed or pre-measured pairs ofleft and right HRTF filters defined on the grid in the three-dimensionalspace (e.g., 200A), where each point of the grid is associated with onepair of left and right HRTF filters. Pairs of HRTF filters may beretrieved by applying an activation function, where the inputs to theactivation function. may include the relative positions anddistance/range between the sound source and the listener, and theoutputs of the activation function can be the determined HRTF databaseindexes to retrieve pairs of HRTF filters defined on grid points. Forexample, in one implementation of the activation function, the inputs tothe activation function can be azimuth angle θ and attitude angle γ,while the outputs are the database index to retrieve a pair of left andright HRTF filters. The retrieved HRTF filters can then be used tofilter the separated sound tracks. For each separated sound track, anactivation function needs to be called to retrieve the correspondingpair of HRTF filters. The values of azimuth angle θ and attitude angle γcan be determined from the user configuration specifications. Forexample, as shown in FIG. 7A, the azimuth angle θ has the values of 0°(vocal), 30° (drum), 180° (bass), and 330° (keyboard) and the attitudeangle γ is 0, then four pairs of the HRTF filters need to be retrievedby the activation function to filter four separated sound tracks,respectively.

As noted below with respect to FIG. 2A and FIG. 2B, the listener (e.g.,202A) and/or the sound source (e.g., 204A) may be moving with angles θand γ changing over time. A sequence of new pair of HRTF filters (e.g.,200B) may then need to be retrieved dynamically in order to output thecorrect binaural sound to virtually represent the sound received by thelistener (e.g., 202A) in the 3D sound space (e.g., 200A). The dynamicretrieval of the HRTF filters may be facilitated by the storage of thefilters as a mesh because a pair of stored HRTF filters may already beassociated with any point on the grid in the 3D space where the listenerand/or sound source(s) may be located during movement. The range R(210A) can be represented by the volume of the filtered sound. Thus, thecloser between the listener to sound source, the louder the soundvolume.

All of the output left audio tracks may then be mixed to generate theleft channel of the binaural sound (e.g., Binaural L), while all theright channels may be mixed to generate the right channel of thebinaural sound (e.g., Binaural R). When both the L and R channels areplayed through earphones or a headset, a listener may experience 3Dbinaural sound and perceive the spatial locations of the sound sourcesin the 3D sound field.

Furthermore, the listener may configure the locations and/or volume ofeach sound source and/or of the listener in the 3D sound field throughthe GUI 110A. Virtually (e.g., in the acoustic virtual reality), thelistener and the sound source(s) may be located in any location withinthe 3D sound field and the volume of each of the sound source(s) may beproportional to the distance from the location of the listener to thelocation of the sound source in the 3D sound field. For example, thesound source location and/or volume may be configured through the GUI110A which may be presented via a user interface device. The userinterface device may be, for example, in the form of a touch screen on asmartphone (FIGS. 7A-7D) or tablet. In one implementation, the virtuallocation of the vocal sound source may be in front of the listener inthe 3D sound field, the drum sound source may be to the front right ofthe listener, the bass sound source may be behind the other soundsources with respect to the listener (e.g. farther away), and “other”instrument (e.g., unidentified sound type or category) may be to thefront left of the listener, with the drum and bass sound sourcesconfigured to be louder and the vocal and “other” sound sourcesconfigured to be quieter by locating the listener (virtual head) nearthe drum and bass (FIG. 7C). The listener may then hear the 3D soundfield, according to the listener’s own configuration, from the binauraloutput (e.g., Binaural L and Binaural R). The listener will hear a soloperformance if placing the virtual head and the instrument in the sameposition (e.g., FIG. 7B).

In one implementation, to generate the binaural output (e.g., BinauralL+R) as shown in FIG. 1A, for each separated sound track associated witha corresponding sound source location, a pair of corresponding HRTFfilters may be selected (e.g., from storage unit 104A) to process (e.g.,by the signal processing unit 106A) the separated sound track into twooutputs: L and R audio. Finally, a mixer (not shown) can mix all of theL and all of the R tracks respectively to output the binaural L, Rsignals. The selection of the corresponding HRTF filters will bediscussed in more detail further below (e.g., see the description ofFIGS. 2 below). If the mixed sound tracks are stereo (two sound tracks),each one of the sound tracks needs to go through the above process togenerate the mixed binaural sound. When both the L and R channels areplayed through earphones or a headset, a listener can experience 3Dbinaural sound and perceive the 3D sound field.

Referring to FIG. 1B, system 100B may include a sound separation unit102B, a 3D signal processing unit 104B, amplifiers 106B, loudspeakers108B, and a 3D sound field configuration unit 110B with a graphical userinterface (GUI) 112B for receiving user input. The sound separation unit102B and the 3D sound field configuration unit 110B may becommunicatively coupled to the signal processing unit 104B. Signalprocessing unit 104B may be a programmable device that may be programmedto implement three-dimensional sound generation according toconfigurations received via the GUI 112B presented on a user interfacedevice (not shown).

In the example of FIG. 1B, the input to sound separation unit 102B isoriginal stereo or mixed sound tracks of mono or stereo signal or audio,while the output from 3D signal processing unit 104B is a set of soundtracks to drive multiple loudspeaker 108B through amplifiers 106B. Eachof the input of mixed tracks or channels may first be separated into aset of separated sound tracks (e.g., for one corresponding sound sourceor type) by the sound separation unit 102B, where each track representsone type (or category) of sound, for example, vocal, drums, bass, orothers (e.g., based on the nature of the corresponding sound source).Each of the separated sound tracks may then be processed by 3D signalprocessing unit 104B to output a single sound track to drive oneloudspeaker 108B through one amplifier 106B, respectively, for eachprocessed sound track. In one implementation, the above-noted processmay be performed in parallel for each of the input mixed sound tracks.All of the output sound tracks may then be played through theloudspeakers 108B (e.g., at different locations in the real world) toform a real world 3D sound field for the listener’s real world location.

As noted above with respect to FIG. 1A, the listener may configure thelocations and/or volume of each sound source and/or of the listener inthe 3D sound field through the GUI 112B. Virtually (e.g., in theacoustic virtual reality), the listener and the sound source(s) may belocated in any location within the 3D sound field and the volume of eachof the sound source(s) may be proportional to the distance from thelocation of the listener to the location of the sound source in the 3Dsound field. For example, the sound source location and/or volume may beconfigured through the GUI 112B which may be presented via a userinterface device. The user interface device may be, for example, in theform of a touch screen on a smartphone or tablet. The listener may thenhear the 3D sound field, according to the listener’s own configuration,from the output of loudspeakers 108B.

An implementation of GUI 110A or GUI 112B may be seen in FIGS. 7A-7Dwhich are described in detail below.

FIGS. 2A-2B illustrate a spatial relationship between a sound source204A and a listener 202A in a three-dimensional space 200A and aselection of HRTF filters 200B for generating 3D sound that reflects thespatial relationship, according to implementations of the presentdisclosure.

A head related transfer function (HRTF) filter (e.g., like those storedin storage unit 104A of FIG. 1A) may characterize how a human listener,with external human ears on a head, at a first specified location in athree-dimensional space receives a sound from a sound source at a secondspecified location in the same 3D space. As sound waves strike thelistener, the size and shape of the head, ears, ear canal, density ofthe head, size and shape of nasal and oral cavities, all transform thesound and affect how it is perceived by the listener, boosting somefrequencies and attenuating others. But the envelop of response spectrummay be more complex than a simple boost or attenuation: it may affect abroad frequency spectrum and/or it may vary significantly from differentsound direction.

With two ears (e.g., binaural hearing), a listener may localize soundsin three dimensions: in range (distance); in direction above and below;and in front and to the rear, as well as to either side. This ispossible because the brain, inner ear and the external ears (pinna) worktogether to make inferences about location. The listener may estimatethe location of a sound source by taking cues derived from one ear(monaural cues), and by comparing cues received at both ears (differencecues or binaural cues). Among the difference cues are time differencesof arrival at each ear and intensity differences at each ear. Themonaural cues come from the interaction between the sound source and thelistener’s human anatomy, in which the original source sound is modifiedby the inner ear and the external ears (pinna) before it enters the earcanal for processing by the cochlea and the brain. These modificationsencode the sound source location, and may be captured via a relationshipbetween the sound source location and the listener’s location. A soundtrack filter based on this relationship is referred to herein as theHRTF filter. Convolution of a sound track with a pair of HRTF filtersconverts the sound to generate binaural signals for left and right earsrespectively, wherein the binaural sound signals (e.g., Binaural L+R ofFIG. 1A) correspond to the real world 3D sound field signals that wouldbe heard at the listener’s location if the source sound were played atthe location associated with the pair of HRTF filters.

A pair of binaural tracks for the left and right ears of the listenermay be used to generate a binaural sound, from mono or stereo, whichseems to come from a particular location in space. A HRTF filter is atransfer function, describing how a sound from a specific location in a3D space will arrive at the listener’s location (generally at the outerend of the listener’s auditory canal). The HRTF filter may beimplemented as convolutional computation in the time domain ormultiplication in the frequency domain to save computation time as shownin FIG. 4 (described more fully below). Multiple pairs of HRTF filtersmay be applied to multiple sound tracks from multiple sound sources togenerate the 3D sound field represented as the binaural sound signals.The corresponding HRTF filters may be selected based on the listener’sconfiguration, i.e., the desired relative locations of sound sources toa listener.

Referring to FIG. 2A, the 3D sound space 200A where sound sources (e.g.,204A) and listener 202A are located may be represented as a grid with apolar coordinate system. The relative location and distance from thelistener 202A to the sound source 204A may be determined according tothree parameters: azimuth angle θ (202B of FIG. 2B), attitude angle γ(204B of FIG. 2B), and radius R (210A).

Referring to FIG. 2B, the corresponding HRTF filters 200B for a listenerat each location in the 3D space 200A may be measured, generated, savedand organized as functions of the polar coordinate system representing3D space 200A. Each HRTF filter 200B (e.g., a pair of left and rightHRTF filters) may be associated with a point on the grid (e.g., the HRTFfilters are stored as a mesh) and each of the grid points may berepresented by two parameters: azimuth angle θ 202B and attitude angle γ204B. Based on a user’s configuration, the system (e.g., 100A of FIG.1A) will know the spatial relationships between each sound source (e.g.,204A) and the listener 202A, i.e., the system will know α 206A, β 208A,and R 210A. Therefore, based on θ = α, and γ = β, the system mayretrieve a corresponding pair of HRTF filters 200B for the left andright ears of the listener (e.g., HRTF_(Right) and HRTF_(Left)), for theseparated sound track associated with the sound source 204A. The soundtrack of the sound source 204A may then be processed (e.g., by signalprocessing unit 106A of FIG. 1A) using the retrieved HRTF filters 200B.The output volume of the generated 3D sound may be a function of radiusR 210A. The shorter the length of R 210A, the louder the output 3D soundvolume.

In an implementation, for multiple sound sources like sound source 204A,the system may repeat the above filter retrieval and filtering operationfor each sound source and then combine (e.g., mix) the filtered soundtracks together for the final binaural output or stereo-kind (superiorto mono) outputs to two loudspeakers.

As noted above with respect to FIG. 1A, the listener 202A and/or thesound source 204A may be moving with angles θ and γ changing over time.A sequence of new pair of HRTF filters 200B may then need to beretrieved dynamically in order to output the correct binaural sound tovirtually represent the sound received by the listener 202A in the 3Dsound space 200A. A new pair of HRTF filters 200B may then need to beretrieved dynamically in order to output the correct binaural sound tovirtually represent the sound received by the listener 202A in the 3Dsound space 200A. The dynamic retrieval of the HRTF filters 200B may befacilitated by the storage of the filters as a mesh because a pair ofstored HRTF filters may already be associated with any point on the gridin the 3D space where the listener and/or sound source(s) may be locatedduring their movement.

FIG. 3 illustrates a system 300 for training a machine learning model308 to separate mixed sound tracks, according to an implementation ofthe present disclosure.

Although music may be recorded on multiple tracks using multiplemicrophones, where each individual track represents each instrument orvocal recorded in a studio, the music streams that consumers most oftenget are mixed into stereo sound. The costs of recording, storage,bandwidth, transmission, and playing of multi-track audio may be veryhigh, so most existing music recordings and communication devices (radioor smartphones) are configured for either mono or stereo sound. Togenerate the 3D soundstage from conventional mixed sound track formats(mono and stereo), the system (e.g., system 100A of FIG. 1A or 100B ofFIG. 1B) may need to separate each mixed sound track into multipletracks where each truck represents or isolates one kind (or category) ofsound or musical instrument. The separation may be performed accordingto a mathematical model and a corresponding software or hardwareimplementation, where the input is a mixed sound track and the output isseparated sound tracks. In an implementation, for stereo input, the leftand right tracks may be processed (e.g., by sound separation unit 102Aof FIG. 1A or sound separation unit 102B of FIG. 2B) jointly orseparately.

Machine learning in this disclosure refers to methods implemented on ahardware processing device that uses statistical techniques and/orartificial neural networks to give computer the ability to “learn”(i.e., progressively improve performance on a specific task) from datawithout being explicitly programmed. The machine learning may use aparameterized model (referred to as “machine learning model”) that maybe deployed using supervised learning/semi-supervised learning,unsupervised learning, or reinforced learning methods.Supervised/semi-supervised learning methods may train the machinelearning models using labeled training examples. To perform a task usingsupervised machine learning model, a computer may use examples (commonlyreferred to as “training data”) to train the machine learning model andto adjust parameters of the machine learning model based on aperformance measurement (e.g., the error rate). The process to adjustthe parameters of the machine learning model (commonly referred to as“train the machine learning model”) may generate a specific model thatis to perform the practical task it is trained for. After training, thecomputer may receive new data inputs associated with the task andcalculate, based on the trained machine learning model, an estimatedoutput for the machine learning model that predicts an outcome for thetask. Each training example may include input data and the correspondingdesired output data, where the data can be in a suitable form such as avector of numerical values or alphanumerical symbols as representationof sound tracks.

The learning process may be an iterative process. The process mayinclude a forward propagation process to calculate an output based onthe machine learning model and the input data fed into the machinelearning model, and then calculate a difference between the desiredoutput data and the calculated output data. The process may furtherinclude a backpropagation process to adjust parameters of the machinelearning model based on the calculated difference.

The parameters for a machine learning model 308 for separating mixedsound tracks may be trained by machine learning, statistical, or signalprocessing technology. As shown in FIG. 3 , the machine learning model308 may have two phases: a training session and a separation session.During the training session for machine learning model 308, audio ormusic recordings of mixed sound may be used as input for featureextraction unit 302 and corresponding separated sound tracks may be usedas targets by separation model training unit 304, i.e., as examples ofdesired separation outputs. The separation model training unit 304 mayinclude a data processing unit including a data normalization/dataperturbation unit 306, and the feature extraction unit 302. The datanormalization normalizes the input training data so that they havesimilar dynamic ranges. The data perturbation generates reasonable datavariations to cover more signal situations than are available in thetraining data in order to have more data for more training. The datanormalization and perturbation may be optional depending on the amountof available data.

The feature extraction unit 302 may extract features from the originalinput data (e.g., mixed sound) in order to facilitate the training andseparation computations. The training data may be processed in the timedomain (raw data), frequency domain, feature domain, or time-frequencydomain through the fast Fourier transform (FFT), short-time Fouriertransform (STFT), spectrogram, auditory transform, wavelets, or othertransforms. FIG. 4 (described more fully below) shows how both soundtrack separation and HRTF filtering may be conducted in a transformeddomain.

The model structure and training algorithms for machine learning model308 may be neural network (NN), convolutional neural network (CNN), deepneural network (DNN), recurrent neural network (RNN), long short-termmemory (LSTM), Gaussian mixture model (GMM), hidden Markov model (HMM),or any model and/or algorithm which may be used to separate soundsources in a mixed sound track. After training, in the separationsession, the input music data may be separated into multiple tracks bythe trained separation model computation unit 310, each separated soundtrack corresponding to one kind of isolated sound. In an implementation,the multiple separated sound tracks may be mixed in different ways fordifferent sound effects through user configuration (e.g., via GUI 110Aof FIG. 1A).

In one implementation, machine learning model 300 may be a DNN or CNNthat may include multiple layers, in particular including an input layer(e.g., training session) for receiving data inputs, an output layer(e.g., separation session) for generating outputs, and one or morehidden layers that each includes linear or non-linear computationelements (referred to as neurons) to perform the DNN or CNN computationpropagated from the input layer to the output layer that may transformthe data inputs to the outputs. Two adjacent layers may be connected byedges. Each of the edges may be associated with a parameter value(referred to as a synaptic weight value) that provide a scale factor tothe output of a neuron in a prior layer as an input to one or moreneurons in a subsequent layer.

Shown in FIGS. 5 (described more fully below), are waveforms andcorresponding spectrograms associated with a mixed sound track of music(e.g., mixed sound input) and separated sound tracks for vocals, drums,bass, and other sound, where the mixed sound track was separated usingthe trained machine learning model 308. The separation computation maybe performed according to the system 400 shown in FIG. 4 .

FIG. 4 illustrates a system 400 for separating and filtering mixed soundtracks using transformed domain sound signals, according to animplementation of the present disclosure.

The training data (e.g., time-domain mixed sound signals) may beprocessed separation unit 404 (like sound separation unit 102A of FIG.1A) in the time domain (e.g., raw data) or a forward transform 402 maybe used so that the training data may be processed in the frequencydomain, feature domain, or time-frequency domain through the fastFourier transform (FFT), short-time Fourier transform (STFT),spectrogram, auditory transform, wavelets, or other transforms. The HRTFfilters 406 (like the ones stored in storage unit 104A of FIG. 1A) maybe implemented as convolutional computation in the time domain or aninverse transform 408 may be used so that the HRTF filters 406 may beimplemented as a multiplication in the frequency domain to savecomputation time. Accordingly, both the sound track separation and theHRTF filtering may be conducted in a transformed domain.

FIGS. 5A-5E illustrate original mixed sound in waveform and spectrogramand the mixed sound separated into vocal, drum, base, and other sound,respectively, according to implementations of the present disclosure.

Shown in FIG. 5A are a waveform and corresponding spectrogram associatedwith a mixed sound track of music (e.g., mixed sound input for system100A of FIG. 1A).

Shown in FIG. 5B are a waveform and corresponding spectrogram associatedwith a separated sound track for vocal sounds from the mixed sound trackof music.

Shown in FIG. 5C are a waveform and corresponding spectrogram associatedwith a separated sound track for drums sounds from the mixed sound trackof music.

Shown in FIG. 5D are a waveform and corresponding spectrogram associatedwith a separated sound track for bass sounds from the mixed sound trackof music.

Shown in FIG. 5E are a waveform and corresponding spectrogram associatedwith a separated sound track for other sounds (e.g., unidentified soundtype) from the mixed sound track of music.

In an implementation of the present disclosure, the mixed sound trackwas separated using the trained machine learning model 308. Theseparation computation may be performed according to the system 400described above with respect to FIG. 4 .

FIG. 6 illustrates far-field voice control of a 3D binaural music system600 with sound separation, according to an implementation of the presentdisclosure.

As an initial matter, microphone array 602 may capture a voice command.The pre-amplifiers/analog to digital converters (ADC) 604 may enlargethe analog signal and/or convert it to a digital signal. Both thepre-amplifier and ADC are optional depending on what kind of microphonesare used in microphone array 602. For example, they may not be needed bydigital microphones.

The acoustic beamformer 606 forms acoustic beam(s) to enhance the voiceor voice command and to suppress any background noise. An acoustic echocanceller (AEC) 608 further cancels the loudspeaker sound (e.g., fromloudspeakers 630) captured by the microphone array 602 using referencesignals. The reference signal may be captured by one or more referencemicrophones 610 near the loudspeakers 630 or from the audio signals(e.g., from configuration/equalizer unit 624) prior to sending tosending them to the amplifier 608 for the loudspeakers 630. The outputfrom the AEC may then be sent to the noise reduction unit 612 to furtherreduce the background noise.

The clean speech is then sent to the wakeup phrase recognizer 614 whichmay recognize a pre-defined wakeup phrase for system 600. The system 600may mute the loudspeakers 630 to further improve voice quality. Theautomatic speech recognizer (ASR) 616 may then recognize the voicecommand, such as a song music title, and then instructs a musicretrieval unit 618 to retrieve the music from a music library 620. In animplementation, the wakeup phrase recognizer 614 and ASR 616 may becombined as one unit. Furthermore, the retrieved music may then beseparated by the sound separation unit 622 that may be like soundseparation unit 102A of FIG. 1A.

A configuration/equalizer unit 624 may then adjust the volume of eachsound source and/or conduct equalization (gain of each frequency band oreach instrument or vocal) of each sound track. Finally, the separatedmusic sound tracks may be played from the loudspeakers 630 (viaamplifier 628) as direct 3D sound as shown in system 100B of FIG. 1B orHRTF filters 626 may be used to process the separated sound tracks inorder to generate binaural sound as shown in system 100A of FIG. 1A.

A few implementations of loudspeaker layouts are shown in FIGS. 8-11 .FIG. 15 illustrates a loudspeaker distribution in a sound bar accordingto an implementation of the present disclosure. In this implementation,each speaker may output sound from a corresponding sound track (e.g.,Track 1, Track 2, ..., Track N). The sound bar may include an array ofloudspeakers or smart speakers. Similarly, FIG. 16 illustrates anexample of loudspeaker distribution in a sound bar with separated stereoplus sound according to an implementation of the present disclosure. Inthis implementation, each pair may include a left speaker for playingthe left sound track and a right speaker for playing the right soundtrack. The 3D sound separation and voice retrieval system can also beapplied to TV and home theaters. The sound from TV, cable, DVD, or CDcan be separated, for example, by ways as shown in FIG. 1B and playedfrom a group of loudspeakers, like a sound bar, or referred to asexamples of loudspeaker arrays as shown in FIGS. 17-18 . FIGS. 17-18illustrate loudspeaker distributions for a TV or a movie theateraccording to implementations of the present disclosure. FIG. 17illustrates a setting in which speakers facing the audience from the TV,and FIG. 18 illustrates a setting in which speakers surrounding theaudience. Each loudspeaker or a pair of loudspeakers (L, R) can play theaudio of one kind of sound, such as vocal or music instrument, or in anycombinations as needed or as user configured.

FIG. 19 illustrates a loudspeaker matrix deployed with a TV or a movietheater according to an implementation of the present disclosure. Asshown in FIG. 19 , another implementation is to place a 2D loudspeakermatrix behind a theater screen or by using a flat, transparent, andscreen top loudspeaker array. Thus, when a person (or sound generatingentity) on the screen is talking, the loudspeakers at locationscorresponding to the person may play the voice sound of the person. Forexample, when watching a band, the music can come from loudspeakers atlocations corresponding to the singer(s) and/or corresponding toinstrumentalist(s) as shown in FIG. 19 . The information of soundcoordinates or locations on the screen can be determined by intelligentalgorithms using artificial intelligence (AI) to recognize the soundlocation from video images and sound type, or by 3D microphones whichrecord sound with sound location information of the sound.

FIGS. 7A-7D illustrate a GUI 700 for user configuration of 3D sound withselected listener positions inside a band formation (7A-7C) and in thefront of the band formation (7D), respectively, according toimplementations of the present disclosure.

In an implementation, the GUI 700 may be configured so that all soundsources (e.g., from a music band on stage) are represented by bandmember icons on a virtual stage and the listener is represented by alistener head icon (wearing headphones to accentuate the position of theleft and right ears) that may be moved freely around the stage by a userof GUI 700. In another implementation, all the icons in FIGS. 7A - 7Dcan be moved freely around the stage though touches by a user of GUI700.

In FIG. 7A, based on the listener head icon being placed at the centerof the virtual stage, the listener may hear the binaural sound and feelthe sound field: the vocal sound is perceived as coming from the front,the drum sounds from the right, the bass sounds from the back, and otherinstruments (e.g., keyboard) from the left.

In FIG. 7B, based on the listener head icon being placed on top of theband drummer icon, the listener may be able to hear the separated drumssolo track.

In FIG. 7B, based on the listener head icon being placed closer to thedrummer and bassist icons, the sounds of drums and bass may be enhanced(e.g., increased volume) while the sound from other instruments (e.g.,vocals and other) may be relatively reduced (e.g., decreased volume),thus, the listener may feel the enhanced bass and beat impact throughconfiguration via GUI 700.

In FIG. 7D another virtual 3D sound field configuration is shown. Inthis configuration, the listener may virtually feel and hear that theband is in the front of her or him even when that is not the case in thereal world music stage recording. The locations of all band member iconsand the listener head icon may be moved anywhere on the GUI 700 displayin order to configure and change the virtual sound field and hearingexperience.

The GUI 700 may also be applicable on a remote to control a TV withdirect 3D sound systems, or other such applications. For example, when auser is watching a move, she or he may move the listener head iconcloser to a vocal icon so that the volume of the voice is increasedwhile the volume of other background sounds (e.g., music) may be reducedso that the user may hear a clearer voice.

FIG. 8 illustrates a system 800 for generating 3D sound with amicrophone array 802, according to an implementation of the presentdisclosure.

The system 800 may be described as a 3D microphone system which maycapture and output 3D and binaural sound directly. As referred toherein, a 3D microphone system may comprise a microphone array systemwhich may capture sounds from different directions together with spatialinformation regarding the location of the sources of the sounds. Thesystem 800 may produce two kinds of outputs: (1) multiple tracks, eachcorresponding to the sound from one direction where each of the multipletracks may drive a group of loudspeakers to represent a 3D sound field;and (2) binaural L and R tracks for earbuds or earphones to virtuallyrepresent the 3D sound field.

Each microphone of microphone array 802 may have their signals processedby a pre-amplifier/ADC unit 804. The pre-amplifiers and analog todigital converters (ADC) may enlarge the analog signal and/or convert itto a digital signal. Both the pre-amplifier and ADC are optional and maydepend on the selected microphone components for microphone array 802.For example, they may not be necessary for digital microphones.

The acoustic beamformer 806 may form acoustic beam patterns pointing todifferent directions or different sound sources, simultaneously, asshown in FIG. 9B. Each of the beams enhance the sound from the “look”direction while suppressing the sound from other directions, to improvea signal to noise ratio (SNR) and to isolate the sound coming from the“look” direction from the sound coming from other directions. A noisereduction unit 808 may further reduce the background noise of thebeamformer outputs if needed. The output from the beamformer maycomprise multiple sound tracks corresponding to sounds coming fromdifferent directions.

In order to generate direct 3D sound, the multiple tracks may drivemultiple amplifiers and loudspeakers to construct a 3D sound field forlisteners.

In order to generate binaural output, the multiple sound tracks may gothrough multiple pairs of selected HRTF filters 810 to convert thespatial sound track to binaural sound. The HRTF filters may be selectedbased on a user’s configuration (e.g., via output audio configurationunit 814) or based on the actual spatial locations of the sound sourcesin the real world. Furthermore, a mixer 812 may then combine the HRTFoutputs to a pair of binaural output for left and right ears,respectively. The final binaural output represents the 3D sound fieldrecorded by the microphone array 802.

Based on the microphone array 802 only having two acoustic beampatterns, pointing to left and right respectively, the microphone arrayworks as a stereo microphone, which is a special case of the 3Dmicrophone.

FIGS. 9A-9B illustrate beam patterns for a 3D microphone 902 and a 3Dmicrophone array 904 with spatial noise cancellation, respectively,according to implementations of the present disclosure.

FIG. 9A shows beam patterns 902 for a 3D microphone 902 which maycapture sound from different directions and spatial informationregarding the sound sources.

FIG. 9B shows a microphone array 904 (e.g., comprising a plurality ofmicrophones 902) configured to capture sounds from two different soundsources A and B with beam patterns A and B formed by respectivebeamformers A and B. The sound captured from sound source A in the“look” direction of one acoustic beam, such as Beam Pattern A, oftenmixes with the sound captured from other directions, such as thedirection of sound source B. In order to cancel the sound coming fromother directions, the 3D microphone array 904 may form another beampattern(s) using the same microphone array 904, such as Beam Pattern B.The sound captured by Beam Pattern B may be used to cancel unwantedmixed in sound captured by Beam Pattern A. Sound from the direction ofsound source B that has been mixed in with sound from Beam Pattern A’s“look” direction may then be cancelled from the output of Beam PatternA. The cancellation algorithm may be provided by an acoustic echocanceller (AEC) unit 906.

FIG. 10 illustrates a conference system 1000 for generatingthree-dimensional sound, according to implementations of the presentdisclosure.

The conference system 1000 may include a signal processing andcomputation unit 1002, a bank 1004 of head related transfer functions(HRTF) filters, a display unit with graphical user interface (GUI) 1006,amplifiers 1008, headset or earphones 1010, and loudspeakers 1012. Thesystem 1000 may be implemented, for example, as software on a user’slaptop, tablet, computer, or smartphone with a connected headset. Thevideo and audio conference, hereinafter referred to as the “conference”,may also be referred to as a teleconference, virtual conference, webconference, webinar, or video conference. One such conference mayinclude multiple local and/or multiple remote attendees. In animplementation, the attendees may be connected by internet and telephonenetworks 1014. In an implementation the conference may be controlled bycloud servers or remote servers via the internet and telephone networks1014.

A user of system 1000 may be one of the attendees of a conference orvirtual concert. She or he is the owner of the laptop, tablet, computer,or smartphone running the conference software with video and audio andpossibly wearing headset 1010. The terms “speakers” or “attendees” referto persons attending the conference. The loudspeakers 1012 may be anydevices which can convert an audio signal to audible sound. Theamplifiers 1008 may be an electronic device or circuit to increase thesignal power to drive the loudspeakers 1012 or the headset 1010. Theheadset 1010 may be headphones, ear caps, or in-ear audio devices.

The input signals (e.g., from the cloud via 1014) may include video,audio and the speaker’s identification (ID). The speaker’s ID mayassociate video and audio input to an attendee who is speaking. Based ona speaker’s ID not being available, a new speaker ID may be generated bythe speaker ID unit 1016 as described below.

The speaker ID unit 1016 may obtain a speaker ID from the conferencesoftware based on the speaker ID used for the speaker’s videoconferencesession. Furthermore, the speaker ID unit 1016 may obtain a speaker IDfrom a microphone array (e.g., microphone array 802 of FIG. 8 or 904 ofFIGS. 9 ). For example, the microphone array beam patterns in FIG. 9B(e.g., beam patterns A and B) may detect the direction of the speakerwith respect to the microphone array. Based on the detected direction,the system 1000 may detect the speaker ID. Still further, the speaker IDunit 1016 may obtain a speaker ID based on a speaker ID algorithm. Forexample, based on a sound track consisting of multiple speaker’s voices,a speaker ID system may have two sessions, training and inference.During training, using available labels, each speaker’s voice is used totrain a speaker dependent model, one model for one speaker. If the labelis not available, the speaker ID system may perform unsupervisedtraining first and then label a voice from the sound track with speakerID, following by supervised training to generate one model per speaker.During inference, given the conference audio, the speaker identificationunit 1016 may use the trained model to process the input sounds andidentify the corresponding speaker. The model may be Gaussian mixturemodel (GMM), hidden Markov model (HMM), DNN, CNN, LSTM, or RNN.

Based on an attendee speaking, a video window associated with theattendee may be highlighted visually in the display/GUI 1006, so theuser knows which attendee of the conference is speaking, e.g., Attendee2 in FIG. 11 described below. From the location of the speaker, forexample, 50 degrees angle from the user, the system 1000 may retrieve apair of corresponding HRTF filters from pre-stored database or memory1004. The signal processing unit 1002 may perform a convolutioncomputation on the input mono signal with the HRTF filters frompre-stored database or memory 1004. The output from the signalprocessing and computation unit 1002 may have two channels of binauralsounds for left and right ears, respectively. A user or attendee maywear the headset unit 1010 in order to hear binaural sound andexperience 3D sound effects. For example, a user that is not looking atthe display 1006 but is wearing the headset 1010 may still perceivewhich attendee is speaking based on the 3D sound so that the user mayfeel as if she or he is in a real conference room.

Based on multiple display/GUIs 1006 and multiple loudspeakers 1012 beingused in a real conference room, each loudspeaker 1012 may be dedicatedto one speaker’s sound in one display/GUI 1006 at one location. In thissituation, the user does not need to use a headset 1010 and she or hemay experience 3D sound from the loudspeakers 1012. The polarity ofloudspeakers can be placed in a home theater, a movie theater, asoundbar, a TV set, a smart speaker, a smartphone, a mobile device, ahandheld device, a laptop computer, a PC, an automobile vehicle oranywhere with more than one loudspeakers or sound generators.

FIG. 11 illustrates a virtual conference room 1100 displayed for a GUI1006 of a conference system 1000 for generating three-dimensional sound,according to implementations of the present disclosure.

The virtual conference room 1100 may have multiple windows (1102-1112)including video of the user and meeting attendees. The locations of thewindows (1102-1112) may be assigned by the conference software (e.g.,running on laptop) or by the user (e.g., via a display/GUI 1006 of FIG.10 ). For example, the user may move the windows (1102-1112) around toarrange the virtual conference room 1100. In an implementation, thecenter of conference room 1100 may include a virtual conference table.

As noted above, Furthermore, the virtual conference room 1100 may beconfigured by the user so that the video windows (1104-1112) of theattendees may be placed virtually anywhere in the virtual conferenceroom 1100 with a mouse, keypad, or touch screen, etc. From the relativelocation of a speaker (e.g., attendee 2) to the user (e.g., angle fromvideo window 1106 of attendee 2 to video window 1102 of the user),related HRTF’s may be selected and applied automatically for attendee 2when they are speaking.

Methods

For simplicity of explanation, the methods of this disclosure aredepicted and described as a series of acts. However, acts in accordancewith this disclosure can occur in various orders and/or concurrently,and with other acts not presented and described herein. Furthermore, notall illustrated acts may be needed to implement the methods inaccordance with the disclosed subject matter. In addition, those skilledin the art will understand and appreciate that the methods couldalternatively be represented as a series of interrelated states via astate diagram or events. Additionally, it should be appreciated that themethods disclosed in this specification are capable of being stored onan article of manufacture to facilitate transporting and transferringsuch methods to computing devices. The term “article of manufacture,” asused herein, is intended to encompass a computer program accessible fromany computer-readable device or storage media.

The methods may be performed by processing devices that may comprisehardware (e.g., circuitry, dedicated logic), computer readableinstructions (e.g., run on a general purpose computer system or adedicated machine), or a combination of both. The methods and each oftheir individual functions, routines, subroutines, or operations may beperformed by one or more processors of the computer device executing themethod. In certain implementations, the methods may be performed by asingle processing thread. Alternatively, the methods may be performed bytwo or more processing threads, each thread executing one or moreindividual functions, routines, subroutines, or operations.

FIG. 12 illustrates a method 1200 for generating three-dimensionalsound, according to an implementation of the present disclosure.

In one implementation, method 1200 may be performed by the signalprocessing units of system 100A of FIG. 1A or subsystem 100B of FIG. 1B.

At 1202, the method includes receiving a specification of athree-dimensional space (e.g., 200A of FIG. 2A) and a mesh of headrelated transfer function (HRTF) filters (e.g., 200B of FIG. 2B) definedon a grid in the three-dimensional space, wherein the three-dimensionalspace is presented in a user interface of a user interface device (e.g.,GUI 110A of FIG. 1A).

At 1204, the method includes determining (e.g., by sound separation unit102A of FIG. 1A) a plurality of sound tracks (e.g., separated soundtracks), wherein each of the plurality of sound tracks is associatedwith a corresponding sound source (e.g., vocal).

At 1206, the method includes representing a listener (e.g., listener202A of FIG. 2A) and the sound sources (e.g., sound source 204A of FIG.2A) of the plurality of sound tracks in the three-dimensional space.

At 1208, the method includes generating, responsive to a userconfiguration (e.g., via GUI 110A of FIG. 1A) of at least one of aposition of the listener or positions of the sound sources in thethree-dimensional space, a plurality of HRTF filters (e.g., 200B of FIG.2B) based on the mesh of HRTF filters (e.g., stored in storage unit 104Aof FIG. 1A) and the positions of the sound sources and the listener inthe three-dimensional space.

At 1210, the method includes applying each of the plurality of HRTFfilters (e.g., 200B of FIG. 2B) to a corresponding one of the pluralityof separated sound tracks to generate a plurality of filtered soundtracks; and

At 1212, the method includes generating the three-dimensional soundbased on the filtered sound tracks.

FIG. 13 illustrates a method 1300 for generating three-dimensionalsound, according to an implementation of the present disclosure.

At 1302, the method includes capturing sound from the plurality of soundsources with a microphone array (e.g., microphone array 802 of FIG. 8 )comprising a plurality of microphones (e.g., microphone 902 of FIG. 9A).

At 1304, the method includes rendering the three-dimensional sound withone or more loudspeakers (e.g., loudspeakers 108B of FIG. 1B).

At 1306, the method includes removing echoes in the plurality of soundtracks with an acoustic echo cancellation unit (e.g., AEC 608 of FIG. 6).

At 1308, the method includes reducing a noise component in the pluralityof sound tracks with a noise reduction unit (e.g., noise reduction unit612 of FIG. 6 ).

At 1310, the method includes processing the plurality of sound trackswith a sound equalizer unit (e.g., configuration/equalizer unit 624 ofFIG. 6 ).

At 1312, the method includes capturing a reference signal with areference sound capture circuit (e.g., reference microphone 610 of FIG.6 ) positioned at proximity to the one or more loudspeakers (e.g.,loudspeakers 630 of FIG. 6 ), wherein the acoustic echo cancellationunit (e.g., AEC 608 of FIG. 6 ) is to remove the echoes based on thecaptured reference signal.

At 1314, the method includes recognizing voice commands with a speechrecognition unit (e.g., speech recognizer 616 of FIG. 6 ).

Hardware

FIG. 14 depicts a block diagram of a computer system 1400 operating inaccordance with one or more aspects of the present disclosure. Invarious examples, computer system 1400 may correspond to any of thesignal processing units/devices described in relation to the systemspresented herein, such as system 100A of FIG. 1A or system 100B of FIG.1B.

In certain implementations, computer system 1400 may be connected (e.g.,via a network, such as a Local Area Network (LAN), an intranet, anextranet, or the Internet) to other computer systems. Computer system1400 may operate in the capacity of a server or a client computer in aclient-server environment, or as a peer computer in a peer-to-peer ordistributed network environment. Computer system 1400 may be provided bya personal computer (PC), a tablet PC, a set-top box (STB), a PersonalDigital Assistant (PDA), a cellular telephone, a web appliance, aserver, a network router, switch or bridge, a computing device invehicle, home, room, or office, or any device capable of executing a setof instructions (sequential or otherwise) that specify actions to betaken by that device. Further, the term “computer” shall include anycollection of computers, processors, or SoC, that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methods described herein.

In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of either a serveror a client machine in server-client or cloud network environments, orit may act as a peer machine in peer-to-peer (or distributed) networkenvironments. The machine may be an onboard vehicle system, wearabledevice, personal computer (PC), a tablet PC, a hybrid tablet, a personaldigital assistant (PDA), a mobile telephone, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein. Similarly, the term “processor-basedsystem” shall be taken to include any set of one or more machines thatare controlled by or operated by a processor (e.g., a computer or cloudserver) to individually or jointly execute instructions to perform anyone or more of the methodologies discussed herein.

Example computer system 1400 includes at least one processor 1402 (e.g.,a central processing unit (CPU), a graphics processing unit (GPU) orboth, processor cores, compute nodes, cloud server, etc.), a main memory1404 and a static memory 1406, which communicate with each other via alink 1408 (e.g., bus). The computer system 1400 may further include avideo display unit 1410, an alphanumeric input device 1412 (e.g., akeyboard), and a user interface (UI) navigation device 1414 (e.g., amouse). In one embodiment, the video display unit 1410, input device1412 and UI navigation device 1414 are incorporated into a touch screendisplay. The computer system 1400 may additionally include a storagedevice 1416 (e.g., a drive unit), a sound production device 1418 (e.g.,a speaker), a network interface device 1420, and one or more sensors1422, such as a global positioning system (GPS) sensor, accelerometer,gyrometer, position sensor, motion sensor, magnetometer, or othersensors.

The storage device 1416 includes a machine-readable medium 1424 on whichis stored one or more sets of data structures and instructions 1426(e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1426 mayalso reside, completely or at least partially, within the main memory1404, static memory 1406, and/or within the processor 1402 duringexecution thereof by the computer system 1400, with main memory 1404,static memory 1406, and processor 1402 comprising machine-readablemedia.

While the machine-readable medium 1424 is illustrated in an exampleembodiment to be a single medium, the term “machine-readable medium” mayinclude a single medium or multiple media (e.g., a centralized, cloud,or distributed database, and/or associated caches and servers) thatstore the one or more instructions 1426. The term “machine-readablemedium” shall also be taken to include any tangible medium that iscapable of storing, encoding or carrying instructions for execution bythe machine and that cause the machine to perform any one or more of themethodologies of the present disclosure or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, andoptical and magnetic media. Specific examples of machine-readable mediainclude volatile or non-volatile memory, including but not limited to,by way of example, semiconductor memory devices (e.g., electricallyprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM)) and flash memory devices;magnetic disks such as internal hard disks and removable disks; andCD-ROM and DVD-ROM disks.

The instructions 1426 may further be transmitted or received over acommunications network 1428 using a transmission medium via the networkinterface device 1420 utilizing any one of a number of well-knowntransfer protocols (e.g., HTTP). Examples of communication networksinclude a local area network (LAN), a wide area network (WAN), theInternet, mobile telephone networks, plain old telephone (POTS)networks, and wireless data networks (e.g., Bluetooth, Wi-Fi, 3G, and 4GLTE/LTE-A or WiMAX networks). The term “transmission medium” shall betaken to include any intangible medium that is capable of storing,encoding, or carrying instructions for execution by the machine, andincludes digital or analog signals or other intangible medium tofacilitate communication of such software.

Example computer system 1400 may also include an input/output controller1430 to receive input and output requests from the at least one centralprocessor 1402, and then send device-specific control signals to thedevice they control. The input/output controller 1430 may free the atleast one central processor 1402 from having to deal with the details ofcontrolling each separate kind of device.

Language

Unless specifically stated otherwise, terms such as “receiving,”“associating,” “determining,” “updating” or the like, refer to actionsand processes performed or implemented by computer systems thatmanipulates and transforms data represented as physical (electronic)quantities within the computer system registers and memories into otherdata similarly represented as physical quantities within the computersystem memories or registers or other such information storage,transmission or display devices. Also, the terms “first,” “second,”“third,” “fourth,” etc. as used herein are meant as labels todistinguish among different elements and may not have an ordinal meaningaccording to their numerical designation.

Examples described herein also relate to an apparatus for performing themethods described herein. This apparatus may be specially constructedfor performing the methods described herein, or it may comprise ageneral purpose computer system selectively programmed by a computerprogram stored in the computer system. Such a computer program may bestored in a computer-readable tangible storage medium.

The methods and illustrative examples described herein are notinherently related to any particular computer or other apparatus.Various general purpose systems may be used in accordance with theteachings described herein, or it may prove convenient to construct morespecialized apparatus to perform method 500 and/or each of itsindividual functions, routines, subroutines, or operations. Examples ofthe structure for a variety of these systems are set forth in thedescription above.

The above description is intended to be illustrative, and notrestrictive. Although the present disclosure has been described withreferences to specific illustrative examples and implementations, itwill be recognized that the present disclosure is not limited to theexamples and implementations described. The scope of the disclosureshould be determined with reference to the following claims, along withthe full scope of equivalents to which the claims are entitled.

1. A three-dimensional sound generation system, comprising: one or moreprocessors of a computing device, communicatively coupled to or embeddedwith a user interface device, the three-dimensional sound generationsystem having capability to: receive one or more sound tracks, each ofthe one or more sound tracks comprising one or more sound sources, eachof the one or more sound sources corresponding to one or more respectivesound categories; receive or determine a first configuration in athree-dimensional space, the first configuration comprising a listenerposition and a computing device location relative to the listenerposition; determine a second configuration comprising a change to atleast one of the listener location or the computing device locationrelative to the listener position; responsive to determining at leastone of the first configuration or the second configuration, generate,using the one or more sound tracks and the second configuration, one ormore channels of sound signals; and provide the one or more channels ofsound signals to drive one or more sound generation devices to generatea three-dimensional sound field that renders a sound effect as if thelistener at the listener position according to the at least one of thefirst configuration or the second configuration perceives the one ormore sound sources originating from the computing device location in thethree-dimensional space.
 2. The three-dimensional sound generationsystem of claim 1, wherein the computing device is one of a smartphone,a smart speaker, a soundbar, a television set, a game console, ahome/theater sound system, a computer, a tablet computer, an automobilevehicle, a headset, earphones, headphones, earbuds, headsets, a helmet,or a cloud server, and wherein the one or more sound generation devicescomprise one or more earphones, headphones, earbuds, headsets, orloudspeakers, and wherein the sound category comprises one or more of avoice sound, a vocal sound, an instrument sound, a car sound, ahelicopter sound, an airplane sound, a vehicle sound, a gunshot sound, afootstep sound, an explosion sound, a sound in a movie, a sound in agame, or an environmental sound.
 3. The three-dimensional soundgeneration system of claim 1, wherein the three-dimensional soundgeneration system has further capability to: responsive to receiving ordetermining the first configuration, generate, using the one or moresound tracks and the first configuration, the one or more channels ofsound signals; and provide the one or more channels of sound signals todrive the one or more sound generation devices to generate thethree-dimensional sound field that renders the sound effect as if thelistener at the listener position according to the first configurationperceives the one or more sound sources originating from the computingdevice location in the three-dimensional space.
 4. The three-dimensionalsound generation system of claim 1, wherein the three-dimensional soundgeneration system has further capability to: responsive to receiving ordetermining the first configuration, present in a configurationenvironment, a representation representing a listener at the listenerlocation; determine a second configuration comprising a change to thelistener location; and present, in the configuration environment, therepresentation representing the listener at the changed listenerlocation.
 5. The three-dimensional sound generation system of claim 1,wherein the three-dimensional sound generation system has furthercapability to: determine, by using one or more sensors, at least one ofa relative location or a movement between the computing device and thelistener; responsive to determining at least one of the relativelocation or the movement between the computing device and the listener,determine the second configuration.
 6. The three-dimensional soundgeneration system of claim 1, wherein the three-dimensional soundgeneration system has further capability to: responsive to determiningthe second configuration, determine a plurality of head related transferfunction (HRTF) filters based on the second configuration comprising thelistener location and the computing device location relative to thelistener position; and generate, using the one or more sound tracks andthe plurality of HRTF filters, the one or more channels of sound signalsto render the three-dimensional sound effect.
 7. The three-dimensionalsound generation system of claim 1, wherein the three-dimensional soundgeneration system has the capability performed by the one or moreprocessors of the computing device.
 8. The three-dimensional soundgeneration system of claim 1, wherein the three-dimensional soundgeneration system has the capability performed by one or more processorsembedded in the sound generation device.
 9. A method comprising:receiving one or more sound tracks, each of the sound tracks comprisingone or more sound sources, each of the one or more sound sourcescorresponding to one or more respective sound categories; receiving ordetermining a first configuration in a three-dimensional space, thefirst configuration comprising a listener position and a computingdevice location relative to the listener position; determining a secondconfiguration comprising a change to at least one of the listenerlocation or the computing device location relative to the listenerposition; responsive to determining the second configuration,generating, using the one or more sound tracks and the secondconfiguration, one or more channels of sound signals; and providing theone or more channels of sound signals to drive one or more soundgeneration devices to generate a three-dimensional sound field thatrenders a sound effect as if the listener at the listener positionaccording to the second configuration perceives the one or more soundsources originating from the computing device location in thethree-dimensional space.
 10. A three-dimensional sound generationsystem, comprising one or more processors, the three-dimensional soundgeneration system having capability to: receive a sound from one or moresound sources, wherein each of the one or more sound sources correspondsto one or more respective sound categories; receive or determine a firstconfiguration in a three-dimensional space, the first configurationcomprising a listener position and one or more sound source locationsrepresenting a listener at the listener location in thethree-dimensional space and the one or more sound sources at the one ormore sound source locations; determine a second configuration comprisinga change to at least one of the listener location or the one or moresound source locations in the first configuration; separate the soundinto one or more sound tracks, wherein each of the one or more soundtrack representing one or more sound categories from the one or morelocations in the three-dimensional space; generate, using the separatedone or more sound tracks and at least one of the first or the secondconfiguration, one or more channels of sound signals; and provide theone or more channels of sound signals to drive one or more soundgeneration devices to generate a three-dimensional sound field thatrenders a sound effect as if the listener at the listener positionaccording to the second configuration listens to the one or more soundsources at the one or more sound source locations in thethree-dimensional space.
 11. The three-dimensional sound generationsystem of claim 10, wherein the computing device is one of a smartphone,a smart speaker, a soundbar, a television set, a game console, ahome/theater sound system, a computer, a tablet computer, an automobilevehicle, a headset, earphones, headphones, earbuds, headsets, aloudspeaker, a helmet, or a cloud server, and wherein the one or moresound generation devices comprise one or more earphones, headphones,earbuds, headsets, or loudspeakers, and wherein the sound categorycomprises one or more of a voice sound, a vocal sound, an instrumentsound, a car sound, a helicopter sound, an airplane sound, a vehiclesound, a gunshot sound, a footstep sound, an explosion sound, a sound ina movie, a sound in a game, an environmental sound, a natural sound, anartificial sound, or a computer-generated sound.
 12. Thethree-dimensional sound generation system of claim 10, wherein thethree-dimensional sound generation system has further capability to:responsive to receiving or determining the first configuration, presentin a configuration environment, representations representing a listenerat the listener location and one or more sound sources at the one ormore sound source locations; and responsive to a user interaction usingthe configuration environment with the representations representing thelistener at the listener location and the one or more sound sources atthe one or more sound source locations, determine the secondconfiguration comprising the change to at least one of the listenerlocation or the one or more sound source locations in the firstconfiguration.
 13. The three-dimensional sound generation system ofclaim 12, wherein the configuration environment comprises one or moregraphic user interfaces, wherein the representations are graphic iconsin the graphic user interface, and wherein the user interactioncomprises an action directed at the graphic representations in thegraphic user interface, wherein the second configuration comprises avirtual change to the one or more sound source locations in the firstconfiguration by the user using the one or more graphic user interface.14. The three-dimensional sound generation system of claim 10, whereinthe three-dimensional sound generation system has further capability to:responsive to receiving or determining the second configuration, selecta plurality of head related transfer function (HRTF) filters based onthe second configuration comprising the listener location and the one ormore sound source locations; and generating, using the separated one ormore sound tracks, the second configuration, and the plurality of HRTFfilters, one or more channels of sound signals; and providing the one ormore channels of sound signals to drive one or more sound generationdevices to generate a three-dimensional sound field that renders a soundeffect as if the listener at the listener position according to thesecond configuration listens to the one or more sound sources at the oneor more sound source locations in the three-dimensional space.
 15. Amethod comprising: receiving a sound generated from one or more soundsources, wherein each of the one or more sound sources corresponds toone or more respective sound categories; determining a firstconfiguration in a three-dimensional space, the first configurationcomprising a listener position and one or more sound source locationsrepresenting a listener at the listener location in thethree-dimensional space and the one or more sound sources at the one ormore sound source locations; determining a second configurationcomprising a change to at least one of the listener location or the oneor more sound source locations in the first configuration; separatingthe sound into one or more sound tracks, wherein each of the one or moresound track representing one or more sound categories from the one ormore locations in the three-dimensional space; generating, using theseparated one or more sound tracks and the second configuration, one ormore channels of sound signals; and providing the one or more channelsof sound signals to drive one or more sound generation devices togenerate a three-dimensional sound field that renders a sound effect asif the listener at the listener position according to the secondconfiguration listens to the one or more sound sources at the one ormore sound source locations in the three-dimensional space.
 16. Athree-dimensional sound generation system, comprising one or moreprocessors, communicatively coupled to one or more sound generationdevices, the three-dimensional sound generation system having capabilityto: receive a sound that is composed of the one or more sound sources,each of the one or more sound sources corresponding to a respectivesound category; obtain a configuration of a three-dimensional spacecomprising one or more listener positions and one or more the soundsource locations; select one or more head related transfer function(HRTF) filters from a predetermined array of HRTF filters based on therelative positions of the sound source locations and the one or moresound listeners in the three-dimensional space; generate, using theselect one or more HRTF filters and the relative positions of the soundsource locations and the one or more sound generation devices in thethree-dimensional space, one or more channels of sound signals; andgenerate a three-dimensional sound field that renders a sound effect asif one or more listeners at the one or more listener positions listen tothe one or more sound sources at the one or more sound source locationsin the three-dimensional space.
 17. The three-dimensional soundgeneration system of claim 16, wherein the three-dimensional soundgeneration system has further capability to: dynamically update theselection of the one or more HRTF filters selected from thepredetermined array of HRTF filters based on an update to the relativepositions of the sound source locations and the one or more soundgeneration devices in the three-dimensional space.
 18. Thethree-dimensional sound generation system of claim 16, wherein the oneor more sound generation devices comprise one or more earphones,headphones, earbuds, headsets, or loudspeakers, and wherein the soundcategory comprises one or more of a voice sound, a vocal sound, aninstrument sound, a car sound, a helicopter sound, an airplane sound, avehicle sound, a gunshot sound, a footstep sound, an explosion sound, asound in a movie, a sound in a game, or an environmental sound.
 19. Thethree-dimensional sound generation system of claim 16, wherein thethree-dimensional sound generation system has further capability to:responsive to receiving or determining the first configuration, presentin a configuration environment, representations representing one or moresound sources at the one or more sound source locations and one or moresound generation devices at the one or more sound generation locations;and responsive to a change in the configuration environment with therepresentations representing the one or more sound sources at the one ormore sound source locations and one or more sound generation devices atthe one or more sound generation locations, determine the secondconfiguration comprising the change to at least one of the one or moresound source locations or the one or more sound generation devices atthe one or more sound generation device locations in the firstconfiguration.
 20. The three-dimensional sound generation system ofclaim 19, wherein the configuration environment comprises a graphic userinterface, one or more files, or one or more computer programs, whereinthe representations are graphic representations in the graphic userinterface, and wherein the change comprises an action directed at thegraphic representations in the graphic user interface.
 21. Thethree-dimensional sound generation system of claim 16, wherein thepredetermined array of HRTF filters comprise HRTF filters defined onnodes of a mesh of grids.
 22. A method comprising: receiving a soundthat is composed of the one or more sound sources, each of the one ormore sound sources corresponding to a respective sound category;obtaining a configuration of a three-dimensional space comprising one ormore listener positions and one or more the sound source locations;selecting one or more head related transfer function (HRTF) filters froma predetermined array of HRTF filters based on the relative positions ofthe sound source locations and the one or more sound listeners in thethree-dimensional space; generating, using the select one or more HRTFfilters and the relative positions of the sound source locations and theone or more sound generation devices in the three-dimensional space, oneor more channels of sound signals; and generating a three-dimensionalsound field that renders a sound effect as if one or more listeners atthe one or more listener positions listen to the one or more soundsources at the one or more sound source locations in thethree-dimensional space.